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PURPOSE 

The  purpose  of  this  report  is  to  familiarize  the  reader  with  the  training 
effectiveness  analysis  (TEA)  process  as  conducted  by  the  TEA  Division  of  the 
TRADOC  Systems  Analysis  Activity,  White  Sands  Missile  Range,  New  Mexico.  The 
report  includes  a  brief  history  of  the  Division,  a  listing  of  all  TEA 
projects  completed  to  date,  a  discussion  of  the  basic  concepts  underlying  the 
TEA  process,  summary  presentations  of  TEA  procedures,  techniques  and  problem 
areas,  and  a  consideration  of  future  developments  and  directions. 

This  is  not  a  "how  to"  report.  The  reader  who  is  planning  to  conduct  a 
training  effectiveness  analysis  should  refer  to  the  TEA  Handbook  (Draft)  for 
procedural  details.  This  report  may  prove  useful  as  a  reference  to  studies 
already  completed  in  certain  areas.  Completed  TEAs  are  available  through  the 
Defense  Technical  Information  Center  (DTIC).  TRASANA  has  a  limited  supply  of 
some  of  the  reports. 
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The  Training  Effectiveness  Analysis  (TEA)  Division  of  the  TRADOC  Systems 
Analysis  Activity  (TRASANA)  has  been  in  existence  for  slightly  less  than  four 
years.  Although  relatively  new  to  the  Army,  the  Division  has  made  signifi¬ 
cant  contributions  toward  the  development  of  more  effective  training 
programs.  Presented  below  is  an  overview  of  the  TEA  Division's  organization, 
history,  and  structure. 

TRASANA— THE  PARENT  ORGANIZATION 

In  July  of  1974,  a  group  of  scientists  and  engineers  associated  with  the 
SAFEGUARD  Systems  Evaluation  Agency  at  White  Sands  Missile  Range  in  New 
Mexico,  were  organized  under  the  Army's  Training  and  Doctrine  Command 
(TRADOC)  as  the  TRADOC  Systems  Analysis  Activity  (TRASANA).  Today,  TRASANA 
employs  over  200  analysts,  both  civilian  and  military,  possessing  a  variety 
of  special  skills  including  physical  scientists,  operations  research 
analysts,  economists,  psychologists,  training  systems  analysts,  statisti¬ 
cians,  mathematicians,  and  engineers.  The  mission  of  TRASANA,  more  fully 
stated  in  TRASANA  Pamphlet  10-1,  is  to  serve  as  an  analytical  center  for 
combat  and  training  developments  and  to  conduct  analyses  necessary  to  support 
the  TRADOC  Commander's  responsibilities  in  the  weapons  system  acquisition 
process.  The  activity  is  divided  into  eight  technical  divisions  and  three 
administrative  and  support  divisions,  each  with  its  own  particular  responsi¬ 
bilities  (see  Figure  1,  TRASANA  Organizational  Chart).  The  analysis 
activities  of  these  divisions  cover  a  broad  array  of  weapon  and  training 
systems.  These  include  cost  and  operational  effectiveness  analyses  (COEA), 
survivability  and  vulnerability  analyses,  validation  of  wargame  simulations, 
development  of  force-on- force  combat  models,  and  training  effectiveness 
analyses  (TEA).  Highlighted  in  the  organizational  chart  are  the  Training 
Effectiveness  Analysis  Division  and  the  Resource  Analysis  Branch,  Special 
Studies  Division.  Although  the  emphasis  of  this  report  is  on  the  activities 
of  the  TEA  Division,  an  integral  part  of  many  TEA  is  cost  analysis  which  is 
conducted  by  the  Resource  Analysis  Branch.  The  organizational  chart  also 
highlights  the  TEA  Management  Agent  who  is  responsible  for  establishing  the 
TEA  priority  list  and  permeating  the  TRADOC  TEA  system  throughout  TRADOC  and 
the  Army. 

THE  TEA  DIVISION 

Prior  to  the  formation  of  the  TEA  Division,  training  analyses  were 
somewhat  dispersed  throughout  TRASANA.  Training  studies  were  conducted  by 
the  Armor,  Artillery,  and  Special  Studies  Divisions.  TRADOC's  recognition  of 
the  importance  of  TEA  and  the  need  for  a  single  agency  to  manage  the  TEA 
process  led  to  the  establishment  of  the  TEA  Division  in  October  1978. 

The  TEA  Management  Agent  is  charged  with  organizing,  directing,  and 
coordinating  the  TRADOC  TEA  system.  The  TEA  Division  conducts  tra-ining 
effectiveness  analyses  in  partnership  with  proponent  centers  and  schools  and 
reviews  TEA  studies  conducted  independently  by  proponents  and  other  TRADOC 
agencies  or  contractors.  The  TEA  Division  conducts  investigations  of  a 
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Figure  1.  TRASANA  Organizational  Chart 
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variety  of  training  subsystems.  The  results  are  provided  to  appropriate 
agencies  to  support  decisions  on  developing  hardware  systems,  modifying 
existing  training  programs,  and  specifying  requirements  for  training  devices. 
The  Division  is  responsible  for  developing  methods  to  measure  effectiveness, 
for  assuring  that  testing  is  properly  designed  and  controlled,  for  developing 
assessment  methods,  and  for  synthesizing  and  interpreting  the  results  of 
analyses.  The  Division  also  provides  consultation  to  other  Army  agencies  in 
the  areas  of  training  equipment,  systems  development  and  measures  of  training 
effectiveness.  In  summary,  the  TEA  Division  serves  as  a  center  of  analytical 
expertise  for  conducting  training  effectiveness  analyses. 

STRUCTURE  OF  THE  TEA  DIVISION 

Currently,  the  TEA  division  is  organized  into  four  branches.  Although 
there  is  some  overlap,  each  branch  serves  as  the  primary  point  of  contact 
(POC)  for  certain  centers  and  schools  as  shown  in  Figure  2. 


Figure  2. 

TEA  Division  Points  of  Contact  for  TRADOC  Schools  and  Centers 


The  TEA  Division  is  an  organization  of  people  with  special  training  and 
abilities.  Across  the  four  branches,  the  levels  and  areas  of  expertise  are 
quite  similar  since  the  general  requirements  for  a  training  analysis  are 
similar  even  though  the  devices  or  systems  under  study  may  be  different. 
Table  A  shows  the  various  areas  of  academic  specialization  possessed  by  the 
members  of  the  Division  and  the  current  number  of  personnel  in  each  area  at 
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each  degree  level.  These  figures  include  seven  military  and  38  civilian 
personnel.  (Currently  five  commissioned  officer  slots  are  not  filled.) 
majority  of  the  TEA  technical  staff  have  academic  training  in  psychology, 
mathematics,  education,  engineering,  and  physics.  The  military  member's 
the  Division  provide  both  analytical  and  military  expertise  a  no  serve  as 
link  between  the  civilian  analysts  and  the  Army. 


TABLE  A 

NUMBER  Of  PERSONNEL,  DEGREE 
ACADEMIC  SPECIALIZATION  W 


AREAS  OF 
V I S I  C'N 


r 


r 


Area  of  Specialisation 

Degree  Engineering  Physics  Psychology  Education  ’-'a  tr>' nat  i  ss 


PhD 

MA 

MS 

MEd 

BA 

BS 

Other 


3 

3 


7  1 

4  4 

1  1 

2 

3  1  1 


2  1 
1 


9 


9 


Total  7  4  12  9  94 
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Each  study  undertaken  by  the  Division  is  conducted  by  an  interdiscipli¬ 
nary  team  made  up  of  those  individuals  who  possess  the  skills,  abilities  and 
knowledge  most  relevant  to  the  problem.  The  team  may  include  an  analyst  from 
the  Resource  Analysis  Branch  of  the  Special  Studies  Division.  The  Resource 
Analysis  Branch  provides  the  coordinated  cost  analysis  which  may  be  required 
for  integration  with  the  training  analysis.  There  are  thirteen  Operations  • 

Research  Analysts  who  provide  cost  analyses  for  TEA  and  for  combat 
developments  studies. 


The  TEA  Division  supports  the  TRADOC  schools  and  centers  (including  » 

integrating  centers)  and  provides  input  to  field  units.  The  TRADOC  centers  ~ 

and  schools  typically  are  the  proponents  for  TEA  even  though  the  analytical 
results  also  support  the  field  units  where  the  data  were  collected.  The 
proponents  and  the  number  of  TEA  completed  for  each  as  of  June  1982  are  shown 
in  Figure  3.  Major  studies  and  proponents  are  listed  in  Table  B,  and  letter 
reports  and  proponents  are  listed  in  Table  C.  ~  * 


4 


9 


The  TEA  Division  usually  becomes  involved  in  a  study  by  one  of  two 
methods.  The  proponent  submits  a  formal  request  for  the  TEA  Division's 
assistance  through  the  Deputy  Chief  of  Stafr  for  Training  (DCST),  TRADOC,  or 
the  proponent  directly  contacts  the  TEA  Division  for  assistance.  The 
Division  in  turn  will  obtain  concurrence  to  perform  the  study  from  the  DCST. 
In  either  case,  the  Division  is  usually  aware  of  the  study  requirement  based 
on  a  list  of  potential  study  areas  maintained  by  the  TEA  Management  Agent. 
This  list  is  derived  from  several  sources.  Projects  needed  during  a  given 
time  frame  are  the  first  source  of  information  provided  to  the  TEA  Management 
Agent.  This  information  is  received  from  the  TRADOC  schools  and  centers  on  a 
recurring  basis.  These  lists  are  subsequently  compared  to  other  documents 
containing  information  on  potential  study  areas  (AR  5-5,  DA  Designated  Major 
or  Non-major  Systems).  The  determination  to  undertake  a  study  is  then  based 
on  the  information  available  to  the  TEA  Management  Agent.  Additional  con¬ 
siderations  include  the  resources  of  the  TEA  Division  needed  to  accomplish 
the  study  in  the  time  frame  desired  by  the  proponent  and  a  subjective 
evaluation  of  the  relative  value  of  a  study. 


To  date,  the  Division  has  conducted  several  different  types  of  TEA  to 
meet  the  needs  of  proponents.  Some  of  these  have  taken  a  very  broad  approach 
to  training  analysis  while  others  have  had  a  more  narrow  focus.  The  scope  of 
the  study  is  determined  by  the  nature  of  the  problem.  Underlying  each  study 
is  a  central  concept  which  provides  direction  to  the  TEA  and  is  applied  in 
whole,  or  in  part,  to  each  TEA  conducted  by  the  Division.  This  central 
concept  is  that  soldier  proficiency,  how  well  someone  performs  his  or  her 
job,  is  determined  by  five  proficiency  factors  and  their  interrelationships. 
One  or  more  of  these  factors  is  analyzed  in  each  TEA. 

The  first  factor  is  the  individual  soldier.  Individuals  vary  along 
several  dimensions  incl udi ng  mental  ability  and  aptitudes,  physical  ability, 
previous  training  and  experience,  motivation,  attitudes,  and  perceptions. 

Each  of  these  can  impact  on  a  soldier's  successful  completion  of  a  training 
program  or  performance  in  a  unit,  and  certain  of  these  factors  may  be  altered 
by  training.  For  example,  analysis  may  find  that  soldiers  with  particular 
aptitudes  are  more  successful  in  completing  a  given  training  program.  To 
maximize  the  effectiveness  of  the  training  only  soldiers  possessing  that 
particular  aptitude  should  be  chosen  as  trainees  or  the  training  should  be 
designed  to  be  compatible  with  soldier  abilities. 

The  second  factor  is  the  trainer  or  instructor.  Knowledgeable, 
effective  trainers  are  critical  to  the  success  of  a  training  program.  Not 
only  must  the  instructor  know  the  subject,  he  must  also  know  how  to  teach 
what  he  knows.  Equally  important  are  his  attitudes  toward,  and  perceptions 
of,  his  role  as  a  trainer  and  what  is  being  trained.  TEAs  frequently  include 
testing  and/or  observation  of  instructors  and  an  assessment  of  their 
attitudes  and  opinions. 
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THE  NUMERALS  SHOW  THE  NUMBER  OF  STUOIES  COMPILED  FCi?  TSE  INDICATED  PROPONENT 


TABLE  B 

MAJOR  REPORTS  COMPLETED 
BY  TEA  DIVISION 


REPORT  ABBREV*  NUMBER  DATE  PROPONENT 

Basic  Rifle  Marksmanship  Cost  and 
Training  Effectiveness  Analysis 
(CTEA) 


REDEYE  Weapon  Systems  Training 
Effectiveness  Analysis  (WSTEA) 
M60A1  Modified  WSTEA 
REDEYE  Weapon  Systems  Army 
Training  Study  (ARTS) 

TEA  Handbook 

Marksmanship  and  Gunnery  Laser 
Device/Infantry  Remoted  Target 
System  Training  Developments 
Study  (TDS) 

Multiple  Launch  Rocket  System  CTEA 
Infantry  Fighting  Vehicle  Initial 
CTEA 

Patriot  Air  Defense  System  CTEA 
VULCAN  Weapons  System  Training 
Subsystem  Effectiveness  Analysis 
(TSEA) 

Cavalry  Fighting  Vehicle  Force 
Development  Test  and  Experimenta¬ 
tion  (FDTE)  Training  Analysis 
Armor  Training  Test  Instruments 
and  Selection  Criteria  Evalua¬ 
tion  Study  TEA 
Plastic  Ammunition  TDS 
Application:  Military  Opns  in 
Urban  Terrain 
Application:  5.56mm  Rifle 
Marksmanship  Sustainment 
Firefinder  TEA  Operator 
Selection  Criterion 
TDS  for  MOULAGES 
Air  Defense  Accessions  TEA 
Near-Term  Scout  Helicopter 
Prel imi nary  CTEA 
CHAPARRAL/REDEYE  TSEA 
Training  Attrition  Problem, 

Institute  for  Military 
Assistance  TSEA 

Ml  (ABRAMS)  Main  Battle  Tank  TEA 
Air  Defense  Accessions  Update 
TDS  Ml  (ABRAMS)  Tank  Unit-Conduct 
of  Fire  Trainer 

Multiple  Integrated  Laser  Engagement 
System  Air  Ground  Engagement 
Simulation/Air  Defense  CTEA 
ELSAP  2000  Tank  Gunnery  Turret 
Trainer  TDS 

Ml  ABRAMS  Tank  Driver  Trainer  TDS 
PERSHING  II  TEA 
UH-60  Flight  Simulator  TDS 
TDS-Bradley  Fighting  Vehicle  Unit- 
Conduct  of  Fire  Trainer 
Corps  Support  Weapon  System- 
Prel iminary  TEA 


8RM 

TR  16-77 

SEP 

77 

USAIC 

RE  WSTEA 

TR  21-77 

NOV 

77 

USAADS 

M60A1 

TR  4 

1-78 

JUN 

78 

USAARMC 

RE  ARTS 

TR  e 

.-78 

OCT 

78 

USAADS 

TEA  Hbk 

N/A 

AUG 

79 

TRADOC 

MAGLAD/ 

IRETS 

TEA 

1-79 

DEC 

79 

USAIC 

MLRS 

TEA 

3-80 

JUN 

80 

USAFAS 

I  FV 

TEA 

4-80 

MAR 

80 

USAIC 

PATRIOT 

TEA 

8-80 

OCT 

80 

USAADS 

VULCAN 

TEA 

23-80 

OCT 

80 

USAADS 

CFV 

TEA 

31-80 

OCT 

80 

USAARMC 

ATS 

TEA 

38-80 

JAN 

81 

USAARMC 

PA: 

MOUT 

TEA 

41-80 

JAN 

81 

CATRADA 

PA; 

5.56mm 

TEA 

41-80 

JAN 

81 

CATRADA 

FIREFINDER 

TEA 

4-81 

JAN 

81 

USAFAS 

MOULAGES 

TEA 

5-81 

81 

'JSAAH5 

ADA  ACC 

TEA 

7-81 

MAR 

r. 

USAADS 

NTSH 

TEA 

10-81 

API; 

1 

1  „ 

CHAP/RE 

TEA 

12-.8! 

TAP  IMA 

TEA 

13-31 

y 

JS  AIM,' 

Ml 

TEA 

37-81 

SEP 

81 

USAARMC 

ADA  Update 

TEA 

40-81 

OCT 

81 

USAADS 

Ml  UCOFT 

TEA 

11-82 

MAR 

82 

USAARMC 

MILES  AGES/ 

AD 

TEA 

12-82 

MAR 

82 

USATSC 

ELSAP 

TEA 

13-82 

MAR 

82 

USAARMC 

Ml  DVR  TNR 

TEA 

15-82 

APR 

82 

USAARMC 

PERSHING 

TEA 

17-82 

MAR 

82 

USAFAS 

UH60FS 

TDS 

19-82 

APR 

82 

USAAVNC 

FV  UCOFT 

TEA 

28-82 

MAY 

82 

USAIC 

CSWS 

TEA 

23-82 

JUN 

82 

USAFAS 

♦These  abbreviations  are  used  throughout  the  text  to  reference  the  report. 
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TABLE  C 

LETTER  REPORTS  COMPLETED 
&v  TEA  DIVISION 


LETTER  REPORT 

ABBREV* 

NUMBER 

DATE 

PROPONENT 

MOS  35H  and  35B  Training  Course 

Selection  Criteria 

35HS35B 

None 

JUL  80 

USALOGC 

Battal ion  Training  Model  Survey 
Determination  of  Prerequisite 

BTMS 

None 

OCT  80 

USAARMC 

ASVAB  Scores  in  Air  Defense 
MOS-Produci ng  Courses 

ASVAB  LR 

LR  42-81 

NOV  81 

USAADS 

MACE  CEP  Training  Effectiveness 
Infantry  Remoted  Target  System 

MACE 

LR  22-82 

MAY  82 

CATRADA 

Qualification  Standards 

IRETS 

LR  25-82 

MAY  82 

USA  IS 

*These  abbreviations  are  used  throughout  the  text  to  reference  the  report. 


The  third  factor  is  the  training  subsystem.  Assessment  of  the  training 
subsystem  includes  determining  whether  the  program  is  designed  to  teach  what 
the  soldier  needs  to  know,  if  adequate  time  is  allowed  for  the  different 
areas  of  instruction,  if  the  training  aids  and  materials  are  adequate  and 
appropriate,  and  if  the  course-embedded  tests  are  valid. 

The  fourth  factor  is  the  hardware  subsystem.  The  basic  question  to  be 
answered  in  this  part  of  the  analysis  is  whether  the  soldier  is  capable  of 
doing  what  the  hardware  demands,  i.e.,  are  soldier  capabilities  well  matched 
to  hardware  demands?  Generally,  this  problem  area  is  addressed  by 
soldier/trainer  questionnaires  and  human  factor  analyses  of  the  interface 
between  the  soldier  and  the  system.  In  some  cases,  the  "hardware"  may  be  a 
new  training  device  or  simulator,  but  the  analysis  is  similar. 

The  fifth  factor  is  the  training  environment.  Analysis  of  the  training 
environment  includes  assessment  of  how  training  is  managed,  to  what  extent 
training  is  emphasized,  to  what  extent  it  is  funded  and  supported,  the 
availability  of  necessary  supplies  and  equipment,  and  how  severely  and 
frequently  training  is  degraded  by  various  "distractors"  (such  as  work 
details,  VIP  shows,  faulty  scheduling,  and  equipment  malfunctions). 

Figure  4  shows  the  interrel ationships  of  these  five  proficiency  factors, 
with  Proficiency  (P)  centrally  located  in  the  area  common  to  all  factors. 

The  figure  suggests  that  with  capable  and  willing  soldiers,  under  the 
guidance  of  effective  instructors,  using  a  well -conceived  program  to  train 
soldiers  on  a  hardware  system  that  can  be  learned  in  an  environment 
relatively  free  of  distractions,  the  result  will  be  effective  training  and 
proficient  soldiers.  Any  degradation  in  any  o.f  these  factors  can  be  expected 
to  produce  a  concomitant  degradation  in  training  effectiveness  and  soldier 
proficiency.  Whether  stated  explicitly  or  implied,  this  five- factor  concept 
is  the  core  of  training  effectiveness  analyses. 
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Figure  4.  Interaction  of  the  Five  Proficiency  Factors 


WHAT  HAVE  WE  DONE? 

The  studies  completed  by  the  TEA  Division  can  be  classified  according  to 
four  major  themes:  evaluation  of  training,  evaluation  of  training  devices, 
evaluation  of  soldier/hardware  interface,  and  evaluation  of  MOS  selection 
criteria.  Some  studies  may  fall  into  more  than  one  of  these  categories. 
Several  studies  include  cost  analysis  and  are  presented  in  this  section  since 
cost  is  an  integral  part  of  determining  efficiency. 

EVALUATION  OF  TRAINING 

The  evaluation  of  training  encompasses  three  separate  levels:  training 
on  new  equipment,  institutional  training,  and  unit  training.  Tables  D,  E, 
and  F  present  the  studies  and  the  MOS  training  evaluated  by  the  Division  for 
each  of  the  three  levels. 
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TABLE  D 

TEA  INCLUDING  ANALYSIS  OF  TRAINING  ON  NEW  EQUIPMENT 


STUDY 

MOS  NO. 

MOS 

IFV 

1  IB 

I nfantryman 

CFV 

19D 

Caval ry  Scout 

PATRIOT 

16T 

PATRIOT  Operator 

24T 

PATRIOT  Maintainer 

Ml 

19K/L 

Tank  Crewmember/Tank  Driver 

45E 

Turret  Mechanic 

63E 

Tracked  Vehicle  Mechanic 

TABLE  E 
TEA  INCLUDING 

ANALYSIS  OF  INSTITUTIONAL  TRAINING 


STUDY 

TYPE  OF  INST  TRNG 

MOS  NO. 

MOS 

BRM 

BCT 

N/A 

N/A 

RE  WSTEA 

AIT 

16P 

ADA  Short  Range 
Missile  Crewmember 

RE  ARTS 

AIT 

16P 

ADA  Short  Range 
Missile  Crewmember 

VULCAN 

AIT 

16R 

ADA  Short  Range 
Gunnery  Crewmember 

CHAP/RE 

AIT 

16P 

ADA  Short  Range 
Missile  Crewmember 

TAP IMA 

Special  Qual 

Course 

N/A 

N/A 

PERSHING 

AIT 

15E 

PERSHING  Missile 
Crewmember 

PERSHING  Officers 
Course 

(SC ) 13C 

PERSHING  Officer 

TABLE  F 
TEA  INCLUDING 
ANALYSIS  OF  UNIT  TRAINING 


STUDY 

MOS  NO. 

MOS 

RE  WSTEA 

16P 

ADA  Short  Range  Missile 
Crewmember 

M60A1 

N/A 

N/A  (Tank  Crews) 

RE  ARTS 

16P 

ADA  Short  Range  Missile 
Crewmember 

VULCAN 

16R 

ADA  Short  Range  Gunnery 
Crewmember 

CHAP/RE 

16P 

ADA  Short  Range  Missile 
Crewmember 

PATRIOT 

25L 

AN/TSQ-73  ADA  Command  & 

Control  System  Operator/ 
Repairer 

PERSHING 

15E 

PERSHING  Missile  Crewmember 

21G 

PERSHING  Electronics 

Materiel  Specialist 

214E0 

PERSHING  Warrant  Officer 

EVALUATION  OF  TRAINING  DEVICES 

Training  devices,  including  simulators,  often  provide  efficient  soldier 
training.  An  effective  training  device  may  be  as  expensive  as  actual 
equipment  yet  still  be  efficient.  Operational  and  support  costs  associated 
with  training  devices  usually  are  less  than  those  of  actual  equipment,  but 
initial  costs  may  be  much  greater.  In  such  cases,  assessing  efficiency 
requires  a  consideration  of  what  a  device  or  simulator  is  capable  of  training 
which  cannot  be  accomplished  with  the  actual  equipment.  An  important  device 
capability  is  fault  insertion.  It  is  seldom  desirable,  and  usually  not 
permitted,  to  "break"  a  tank,  helicopter,  or  other  piece  of  equipment  so  that 
soldiers  can  be  trained  to  fix  it.  It  is  equally  undesirable  to  create 
emergency  situations,  which  may  even  be  life-threatening,  for  the  purposes  of 
training.  When  a  trainee  "crashes"  in  a  flight  simulator,  he  may  be 
embarrassed  and  feel  a  little  "shook",  but  he's  alive  and  probably  much  wiser 
from  the  experience.  In  the  field,  safety  and  range  restrictions  often 
prevent  unit  training  on  the  actual  weapon  system,  and  the  use  of  a  training 
device  is  the  only  alternative.  The  development  of  plastic  ammunition  (see 
the  PA  studies)  was  a  response  to  such  restrictions. 

Of  course,  there  are  numerous  cases  In  which  a  device  or  simulator  is 
not  only  training  effective  but  results  in  significant  cost  savings.  For 
example,  the  expense  of  firing  live  missiles  and  destroying  target  aircraft 
in  training  REDEYE  soldiers  Is  prohibitive,  so  training  devices  are  required. 
When  training  devices  are  effective  and  much  less  costly  than  using  the 
actual  equipment,  several  devices  can  be  acquired,  thereby  resulting  In  a 
greater  number  of  soldiers  being  trained  and/or  more  frequent  training. 

II 
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There  are  potential  drawbacks  in  the  use  of  devices  and  simulators  which 
TEA  analysts  must  assess.  A  soldier  may  become  proficient  at  "playing  the 
game"  rather  than  acquiring  the  skills  he  needs  for  his  job,  i.e.,  knowing 
how  to  "roll  up"  a  big  score  by  realizing  what  the  simulator  can  or  cannot 
do.  There  also  is  the  possibility  of  negative  training  transfer,  or  less 
proficient  performance,  as  a  result  of  a  poorly  designed  simulator. 

In  summary  there  are  three  basic  questions  to  address  in  evaluating  j| ' 

training  devices/simulators:  (1)  Can  it  be  used  to  train  the  necessary 
skills?,  (2)  Is  it  cost  efficient?,  and  (3)  Are  any  bad  habits  learned? 

Table  G  lists  the  TEA  studies  which  include  an  evaluation  of  training 
devices. 


TABLE  G 

TEA  INCLUDING  AN 
EVALUATION  OF  TRAINING  DEVICE 


STUDY 

TRAINING  DEVICE 

■r 

MAGLAD/IRETS 

Marksmanship  and  Gunnery  Laser  Device,  Infantry  Remoted 

CFV 

Target  System 

Fire  Control  Combat  Simulator  (FCCS) 

•  * 

PATRIOT 

Troop  Proficiency  Trainer  (TPT) 

*  ’ 

VULCAN 

VULCAN  Training  System  (VTS) 

W' 

P/A:  MOUT 

P/A:  5.56mm 

Plastic  Ammunition 

Plastic  Ammunition 

MOULAGES 

Moul age 

CHAP/RE 

AN/TSQ-T3  Monitoring  Set 

Ml  UCOFT 

M30  Training  Missile 

Ground  Observer  Aircraft  Recognition  (GOAR)  Kit 

Radio  Controlled  Miniature  Aerial  Target  (RCMAT) 

CHAPARRAL  Television  Trainer 

Ml  Tank  Unit-Conduct  of  Fire  Trainer 

* 

• 

*•  .  *•-  -v 

MILES  AGES/AD 

Multiple  Integrated  Laser  Engagement  System 

ELSAP 

ELSAP  2000  Tank  Gunnery  Turret  Trainer 

' 

UH60FS 

Flight  Simul ators--Camera  Model  System  (CMS),  Digital 

» 

Ml  DVR  TNR 

Image  Generation  (DIG) 

Ml  ABRAMS  Tank  Driver  Trainer 

-  -  r*  4 

FV  UCOFT 

BRADLEY  Fighting  Vehicle  Unit  -  Conduct  of  Fire  Trainer 

IRETS 

Infantry  Remoted  Target  System 

'■  ■; 

MACE 

Battalion  Level  Wargame  Simulator 

9  ' 

EVALUATION  OF  SOLDIER/HARDWARE  INTERFACE 

The  evaluation  of  the  soldier/hardware  interface  is  an  important  part  of 
many  studies,  particularly  TEA  on  new  or  developing  systems.  If  a  system  is 
not  operating  at  optimum  performance  levels,  it  is  necessary  to  determine  the 
extent  to  which  the  problem  is  due  to  the  soldier,  the  hardware,  or  the  two 
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in  combination  in  the  operational  environment.  Problems  associated  with  the 
soldier  may  be  approached  by  modifying  the  training  program.  Problems  due  to 
the  hardware  may  require  modifying  the  equipment.  Although  modern  technology 
generally  has  simplified  equipment  operation,  even  of  complex  machines, 
equipment  design  deficiencies  are  not  uncommon.  Design  deficiencies  may 
adversely  affect  training  and/or  reduce  soldier  proficiency.  This  type  of 
analysis  requires  the  expertise  of  human  factors  engineering.  Table  H  lists 
the  reports  which  have  addressed  the  soldier/hardware  interface. 


TABLE  H 

STUDIES  ADDRESSING  SOLDIER/HARDWARE 
INTERFACE 


STUDY 


IFV 

MLRS 

PATRIOT 

VULCAN 

CFV 

NTSH 

CHAP/RE 

Ml 

UH60FS 


EVALUATION  OF  SOLDIER  SELECTION  CRITERIA 

Another  major  emphasis  in  the  TEA  reports  is  the  evaluation  of  soldier 
selection  criteria.  For  this  analysis,  course  records  are  obtained  from  the 
TRADOC  schools,  while  Armed  Services  Vocational  Aptitude  Battery  (ASVAB) 
scores  are  usually  obtained  from  the  school,  the  Military  Personnel  Center 
(MILPERCEN)  Alexandria,  VA  or  the  Defense  Manpower  Data  Center  (DMDC) 
Monterey,  CA.  Selection  criteria  analyses  are  usually  required  when  an 
attrition  problem  is  evident  in  an  existing  course.  Another  situation  which 
warrants  this  analysis  is  when  a  new  system  is  being  developed  and  the 
necessary  qualifications  of  soldiers  to  operate/maintain  the  system  are  not 
known.  The  studies  analyzing  the  MOS  selection  criteria,  and  the  MOS  are 
shown  in  Tabl e  I . 

COST  ANALYSIS 

The  efficiency  of  a  training  program  includes  a  consideration  of  both 
training  effectiveness  and  training  costs.  Cost  analyses  may  be  provided  as 
input  to  the  evaluation  of  a  training  program  for  a  new  weapon  system,  in  the 
acquisition  of  a  simulator  or  training  device,  or  when  comparing  two  or  more 
training  alternatives  with  a  current  training  program. 


TABLE  I 
TEA  INCLUDING 

AN  ANALYSIS  OF  SOLDIER  SELECTION  CRITERIA 


STUDY 

MOS  NO. 

MOS 

,  | 

MLRS 

T3B 

Cannon  Crewmember  | 

15D 

LANCE  Missile  Crewmember  i 

PATRIOT 

25L 

AN/TSQ-73  ADA  Command  and  Control 
System  Operator 

ATS 

19E 

M48-M60A1/A3  Armor  Crewmember 

19F 

Tank  Driver 

45N 

Tank  Turret  Mechanic 

63C 

Track  Vehicle  Mechanic 

FIREFINDER 

13E 

Cannon  Fire  Direction  Specialist 

13F 

Fire  Support  Specialist 

ADA  ACC 

All  ADA  MOS 

TAPIMA 

N/A 

(Special  Forces) 

ADA  Update 

All  ADA  MOS 

PERSHING 

1 5  £ 

PERSHING  Missile  Crewmember 

21G 

PERSHING  Electronics  Materiel 

Sped  al  i  st 

ASVAB  LR 

All  ADA  MOS 

35H  &  35B 

35H 

Cal  ibration  Special ist  1 

35B 

Electronic  Instrument  Repairer 

Because  training  programs  and  systems  must  compete  with  other  Army 
programs  for  scarce  resources,  many  training  studies  are  incomplete  without  a 
cost  analysis.  The  decision  as  to  whether  a  cost  analysis  is  required  should 
be  made  in  the  initial  planning  phase  of  the  study  and  must  be  based  on  the 
issue  facing  the  decision  maker.  A  cost  analysis  probably  will  be  necessary 
under  the  following  conditions: 

o  When  resource  requirements  of  any  of  the  study  alternatives  are 
likely  to  be  large 

o  When  resource  requirements  are  likely  to  vary  significantly  among 
the  alternatives 

o  When  the  options  available  to  the  decision  maker  are  likely  to  affect 
resources  available  to  the  program  being  studied 

Table  J  lists  the  studies  which  include  a  cost  analysis. 

HOW  DO  WE  DO  IT? 

As  stated  previously,  training  effectiveness  analyses  address  one  or 
more  of  the  five  proficiency  factors.  Essential  to  the  process  of  assessment 
is  data  collection  through  measurement  and  data  analyses  by  means  of 
statistics.  Training  effectiveness  analyses  require  many  different  types  of 
measures  and  statistical  procedures. 
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TABLE  J 

TEA  INCLUDING  A  COST  ANALYSIS 


STUDY 


BRM 

RE  ARTS 
IFV 
MLRS 
PATRIOT 
CHAP/RE 
Ml  DVR  TRN 
PA:  MOUT 
PA:  5.56mm 
MOULAGES 
Ml  UCOFT 
ELSAP 
UH60FS 


STANDARDIZED  TESTS 


Standardized  tests  are  tests  intended  for  widespread  use  to  assess 
general  attributes,  i.e.,  abilities  or  traits  possessed  to  some  degree  by 
virtually  everyone.  Standardized  tests  are  used  in  TEA  to  determine  sample 
representativeness,  to  develop  selection  criteria,  to  assess  the  appropriate 
ness  of  training  materials,  and  to  assess  soldier  characteristics. 

The  following  list  shows  some  of  the  standardized  tests  which  have  been 
used  frequently  by  the  TEA  Division  and  a  brief  statement  of  the  purpose  of 
each  test. 


o 


o 


o 


o 


Select  Adult  Basic  Learning  Exam  (SelectABLE)  --  Measure  word  and 
number  knowledge;  estimate  Reading  Grade  Level  (RGL). 


Gates  MacGinitie  Reading  Test  --  Measure  reading  vocabulary, 
comprehension  and  RGL. 

Lynn  Achievement  Motivation  Scale  (Lynn)  --  Compare  achievement 
motivation  among  groups. 

Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  --  Measures  of 
general  aptitude  areas  which  are  related  to  certain  military 
occupational  fields.  These  aptitude  areas  (composites)  are: 


General  Technical  (GT) 
General  Maintenance  (GM) 
Electronics  Repair  (EL) 
Combat  (CO) 

Mechanical  Maintenance  (MM) 


Field  Artillery  (FA) 
Operators  and  Food  (OF) 
Skilled  Technical  (ST) 
Clerical  (CL) 
Surveillance  & 
Communications  (SC) 
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o  Armed  Forces  Qualification  Test  Percentile  Score  (AFQT)  --  A  broad 
measure  of  intellectual  ability. 

Listing  1  is  a  reference  of  reports  in  which  these  specific  tests  were  used 
and  is  provided  at  the  conclusion  of  this  report. 

SPECIAL  PURPOSE  TESTS 

In  addition  to  standardized  tests,  it  is  frequently  necessary  for  the 
analysts  to  develop  special  purpose  tests  and  surveys  to  answer  specific 
questions  posed  in  a  particular  study.  Two  types  of  special  purpose  tests 
are  used  frequently  in  the  TEA  process:  written  proficiency  tests,  also 
called  skills  and  knowledge  (S/K)  tests;  and  performance  proficiency  tests, 
better  known  as  hands-on  tests  (HOT). 

Written  proficiency  tests  are  used  to  assess  the  soldier's  knowledge  and 
understanding  of  what  he  has  been  taught.  They  are  particularly  useful  for 
evaluating  the  depth  of  a  soldier's  understanding  and  for  testing  his 
knowledge  of  how  to  do  something.  Written  proficiency  tests  typically  are 
paper-and-pencil  tests  with  a  multiple-choice,  true-false,  and/or  short 
answer  format.  The  majority  of  the  studies  in  the  TEA  Division  have  included 
a  written  proficiency  test  (see  Listing  2). 

Performance  proficiency  tests  require  a  soldier  to  actually  perform 
tasks  he  was  trained  to  do.  The  test  may  be  comprehensive,  i.e.,  include  all 
tasks  trained  or  may  consist  of  selected  representative  tasks.  Performance 
profi-ciency  tests  are  often  scored  on  a  "Go  or  No-Go"  basis.  Either  the 
soldier  performs  the  task  correctly  or  he  does  not.  The  score  is  assigned  by 
the  test  administrator  who  may  be  a  TEA  analyst,  an  instructor,  or  another 
trained  data  collector. 

Written  and  performance  proficiency  tests  provide  the  most  direct  and 
objective  data  by  which  a  training  program  can  be  evaluated  and  serve  as  the 
primary  measures  of  training  effectiveness  (MOTE).  For  the  most  part, 
proficiency  tests  are  developed  (with  the  assistance  of  subject  matter 
experts),  administered,  scored,  and  interpreted  by  analysts  of  the  TEA 
Division. 

SPECIAL  PURPOSE  SURVEYS 

In  addition  to  performance  tests,  soldier  and  trainer  attitudes  and 
perceptions  also  provide  useful  information  in  the  TEA  process.  Typically, 
these  data  are  obtained  by  various  types  of  surveys.  These  surveys  address 
general  attitudes  and/or  specific  attitudes  toward  hardware  and  training. 

General  attitudes  are  assessed  for  a  number  of  reasons.  They  serve  as 
useful  indicators  of  general  motivation  and  can  be  used  to  control  for 
differences  in  training  and  proficiency  resulting  from  motivation  and  general 
outlook  on  training,  the  Army,  and  particular  weapon  systems.  While 
attitudes  do  not  necessarily  correlate  directly  with  measured  proficiency, 


they  do  offer  important  and  useful  insights  into  the  interpretation  of 
proficiency  scores  and  results  of  training.  When  carefully  measured  and 
interpreted,  they  serve  as  a  valuable  "temperature  check"  on  morale  and 
motivation. 

The  usual  procedure  of  attitude  assessment  is  to  have  the  soldier 
indicate  the  extent  to  which  he  agrees  or  disagrees  with  a  particular 
statement.  The  most  frequent  areas  of  general  attitude  assessment  are 
attitudes  toward  the  United  States,  the  Army,  peers,  superiors,  and  the 
soldier's  unit.  In  addition,  assessment  is  made  of  more  specific  attitudes 
toward  training,  trainers,  and  hardware.  Listing  3  shows  the  studies  which 
included  general  attitude  surveys. 

Another  frequently  used  survey  is  an  assessment  of  soldier  perceptions 
of  the  tasks  related  to  his  job.  It  is  important  to  determine  which  tasks 
are  the  most  important  to  the  successful  completion  of  a  job  or  mission  and 
whether  these  critical  tasks  receive  corresponding  emphasis  in  training.  To 
determine  the  extent  to  which  training  addresses  actual  job  requi rements , 
several  TEA  have  included  a  "Task  Frequency,  Criticality,  and  Performance 
Survey"  (which  may  be  expanded  to  include  perceptions  of  task  difficulty  and 
task  training  requirements).  Constructing  these  surveys  may  be  difficult  for 
a  developing  system  because  task  lists  often  are  not  available.  The  studies 
including  these  surveys  are  in  Listing  4  at  the  end  of  the  report. 

Another  source  of  information  in  TEA  is  demographic  surveys  which  yield 
background  and  descriptive  information  on  the  study  participants,  e.g.,  age, 
rank,  educational  level,  and  MOS.  This  information  is  used  to  describe  the 
test  subjects,  determine  if  the  sample  is  representative,  ana  determine  if 
demographic  information  is  related  to  performance  measures.  Another  use  of 
demographic  information  is  to  compare  two  or  more  study  groups  to  insure 
their  approximate  equivalence  in  terms  of  performance  related  variables. 

ADDITIONAL  PROCEDURES  AND  ANALYSES  IN  TEA 

There  are  still  other  procedures  and  analyses  used  in  certain  studies 
other  than  the  tests  and  surveys  discussed  above.  In  some  instances,  (see 
studies  indicated  in  Table  K),  it  is  possible  to  carry  out  force-on-force 
simulations  in  which  observed  proficiency,  usually  as  a  result  of  training, 
is  put  into  a  computer  model  of  a  simulated  combat  engagement.  The  results 
of  the  simulation  approximate  actual  combat  outcomes  given  the  measured 
effectiveness  of  the  training  program. 

Many  TEA  include  human  factors  analyses.  Observations  are  made  as  to 
where  the  soldier-weapon  interface  presents  problems.  For  example,  it  was 
noted  in  the  REDEYE  WSTEA  study  that  soldiers  were  required  to  memorize  a 
matrix  of  28  numbers  to  successfully  engage  a  target.  Subsequent  to  the  TEA, 
the  matrix  was  revised  and  made  simpler  to  apply.  Such  observations,  with 
recommended  improvements,  are  quite  common  in  the  TEA  process. 
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TABLE  K 

STUDIES  INCLUDING  FORCE-ON-FORCE  SIMULATIONS 


COST  ANALYSIS 

The  cost  analysis  associated  with  a  TEA  is  aimed  at  the  evaluation  of 
economic  considerations  on  which  decisions  will  be  made.  It  typically 
follows  the  same  analytical  process  that  is  inherent  in  any  cost  analysis. 

The  costing  approach  is  conceptually  and  methodologically  similar  to  the 
approach  normally  undertaken  for  Cost  and  Operational  Effectiveness  Analysis 
(COEA).  The  procedure  for  a  cost  analysis  is  as  follows: 

o  Define  objective 

o  Define  alternatives 

o  Formulate  assumptions 

o  Develop  methodology  for  estimating  cost 

o  Determine  quantities  costs 

o  Perform  detailed  analysis 

o  Analyze  uncertainties 

o  Write  report 

The  difference  between  the  TEA  and  COEA  approach  is  that  the  TEA  focuses  on 
the  training  subsystem,  while  the  COEA  treats  the  entire  materiel  system. 

The  total  life  cycle  costs  of  alternative  training  subsystems  are  required  to 
establish  a  relative  cost  relationship.  This  total  cost  is  composed  of  two 
elements  -  The  cost  of  acquisition  (nonrecurring)  and  cost  of  ownership 
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(recurring)  with  ownership  cost  being  the  sum  of  all  operating  and  support 
costs.  Therefore,  the  analyst  must: 

o  Distinguish  between  recurring  and  nonrecurring  costs 

o  Insure  that  the  costs  of  the  base  case  and  each  alternative  are  fully 
captured  and  evaluated 

o  Insure  that  all  costs  of  system  support  are  included 

o  Display  total  costs  in  such  a  manner  as  to  allow  valid  comparison  of 
al ternati ves 

STATISTICS 

TEA  require  collection  of  several  different  types  of  data,  including 
demographic  information,  proficiency  tests,  attitude  surveys,  questionnaires, 
simulation  outcomes,  human  factors  analyses,  and  cost  analyses.  Once 
compiled,  these  data  must  be  organized  and  statistically  evaluated.  The 
various  statistical  procedures  used  in  TEA  for  evaluating  differences  between 
groups  and  determining  the  extent  to  which  two  or  more  variables  are  related 
are  shown  in  Listings  5  and  6.  Exactly  which  procedure  is  used  is  determined 
by  the  question  to  be  answered  and  the  nature  of  the  data. 

WHAT  HAVE  HE  LEARNED  ABOUT 
TRAINING  EFFECTIVENESS? 

The  Army's  approach  to  training  is  based  on  sound  concepts,  and  a 
concentrated  effort  to  apply  these  concepts  is  evident.  The  Instructional 
System  Development  (ISD)  procedure  and  the  use  of  criterion  referenced  te-.f.s 
are  good  examples  of  the  Army's  attempt  to  maximize  training  effectiveness. 

In  fact,  the  existence  of  the  TEA  Division  and  the  requirement  for  training 
effectiveness  analyses  are  evidence  of  the  Army's  commitment  to  develop 
quality  training  systems. 

Since  a  primary  purpose  of  the  TEA  process  is  to  optimize  training 
effectiveness  and  prevent  training  deficiencies,  this  section  of  the  report 
focuses  on  problem  areas  that  have  surfaced  in  TEA  projects  to  date.  These 
problems  are  discussed  in  terms  of  the  five  proficiency  factors  presented  in 
the  section  "What  Do  We  Do?".  The  Division  has  looked  at  several  training 
systems,  devices,  and  MOS.  Some  trends  have  emerged  across  the  various 
studies. 

THE  SOLDIER 

In  several  studies  soldier's  attributes  were  found  to  be  directly 
related  to  proficiency.  Army  soldiers  possess  generally  positive  attitudes. 
They  want  to  learn,  take  pride  in  their  units  and  work,  and  are  motivated  to 
do  a  good  job. 
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Soldiers'  attitudes  occasionally  have  been  correlated  with  performance 
and  have  proven  a  valuable  source  of  general  information  on  training  and 
hardware  subsystems.  Less-than-posi ti ve  attitudes  may  result  from  how 
training  is  conducted  (as  in  the  TAPIMA  study),  equipment  malfunctions  (MILES 
AGES/AD),  and  equipment  nonavailability  (PATRIOT).  In  addition  to  thinking 
of  how  training  affects  individual  attitudes,  the  effects  of  collective 
negative  attitudes  on  the  success  of  a  training  program  and  morale  in  general 
also  need  to  be  considered. 

THE  TRAINER 

In  TEA  where  the  effectiveness  of  the  trainer  was  addressed,  it  was 
found  that  most  trainers  are  qualified.  Some  i nstructor-rel ated  problems  at 
AIT  and  in  the  units  have  been  documented.  Unit  trainers  are  often 
transferred  from  one  system  to  another,  with  no  formal  training  on  the  new 
system  (CHAP/RE  and  PERSHING  studies).  One  study  found  instructors  to  be 
knowledgeable  in  the  subject  matter,  but  several  were  unskilled  in  "how  to 
trai n" . 

In  the  Ml  DVR  TNR  report,  it  was  found  that  none  of  the  six  contractor 
trained  instructors  passed  a  performance  proficiency  test.  In  other  words, 
they  could  not  perform  the  tasks  required  to  operate  the  training  device. 

Due  to  frequent  equipment  malfunctions,  it  could  not  be  determined  whether 
their  inability  to  perform  the  tasks  was  due  to  a  training  problem  or  an 
equipment  problem.  Fortunately,  there  was  time  for  additional  instructor 
training  and  equipment  repairs  before  the  course  began. 

Not  all  trainer  problems  are  related  to  instructor  qualifications.  It 
is  a  common  occurrence  that  the  student-to- i nstructor  ratio  exceeds  that 
specified  for  the  course  (see  the  CHAP/RE  and  FIREFINDER  reports).  In  the- 
CHAP/RE  study,  the  student-instructor  ratio  was  18  to  1.  The  number  of 
authorized  instructors  was  35,  but  only  15  were  assigned  and  even  fewer  were 
present  for  duty.  In  such  cases,  the  number  of  instructors,  not  the  quality, 
is  a  major  problem. 

THE  TRAINING  SUBSYSTEM 

To  evaluate  a  training  program,  what  the  trainee  is  expected  to  learn 
and  the  desired  level  of  proficiency  must  be  known.  This  requires  developing 
a  list  of  tasks  to  be  trained  and  the  expected  performance  standards  for  each 
task.  A  common  source  of  task  lists  is  the  soldier's  manual  ( SM )  which 
specifies  the  critical  tasks,  conditions,  and  performance  standards  at  each 
skill  level  for  each  MOS. 

In  the  ATS  study,  it  was  found  that  the  program  of  instruction  (POI)  did 
not  correspond  to  the  SM  task  list.  A  similar  discrepancy  between  the  POI 
and  SM  was  found  in  the  Ml  study  of  new  equipment  training  (NET).  An 
additional  finding  of  the  Ml  study  was  that  the  results  of  course-embedded 
tests  were  not  an  accurate  reflection  of  soldier  proficiency,  i.e.,  the  tests 
lacked  validity. 
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Although  not  consistent  across  all  studies,  numerous  other  problems 
related  to  training  subsystems  have  been  found,  including: 

o  Insufficient  "hands-on"  training 

o  Too  much  emphasis  on  administrati ve,  or  "i rrel evant" ,  training 

o  Lack  of  individual  attention  given  the  trainee 

o  Unnecessary  stress  placed  on  trainees  during  ski  1 1 -acqui si tion 
phase  of  trai ni ng 

o  Unrealistic  training  (reduced  firing  ranges  or  insufficient 
number  of  targets  presented  as  in  the  M60A1  report) 

o  Failure  to  adhere  to  the  ISD  model 

o  Insufficient  training  time 

o  Failure  to  adhere  to  the  requirements  of  a  training  program 

o  Insufficient  testing  within  a  POI  (CFV,  Ml,  ATS) 

o  Inappropriately  written  course  material  for  reading  ability  of 
soldiers 

Perhaps  the  most  significant  general  problem  related  to  training  is  the 
nonstandardized  unit  training.  Consider  this  synopsis  of  unit  training: 

"Unit  training  programs  are  sporadic,  ineffective,  and  are  generally 
characterized  by  high  expenditure  of  limited  resources  without  an 
appreciable  increase  in  mission  capability  and/or  individual  skills."  - 
NTSH  PTEA,  April  81. 

For  those  units  evaluated  in  the  TEA  reports,  nonstandardized  unit 
training  seemed  to  be  widespread  (RE  ARTS,  VULCAN,  CHAP/RE,  NTSH,  PERSHING). 
Units  may  fail  to  adhere  to  training  schedules  or  lack  an  organized  training 
schedule  altogether.  Most  unit  training  is  test-directed,  i.e.,  occurs  in 
preparation  for  some  evaluation  exercise.  The  standards  and  objectives  of 
the  various  tests  may  be  inconsistent  and  sometimes  conflicting  with  each 
other.  Command  emphasis  is  often  oriented  toward  training  to  pass  the  next 
test  and  not  necessarily  toward  complete,  well-balanced  training.  These 
problems  are  prevalent  in  the  Army  and  have  a  marked  impact  on  the 
effectiveness  of  unit  training. 

It  has  been  documented  (PATRIOT)  that  more  training  does  not  improve 
proficiency  when  the  training  is  inadequate  or  poor.  "Practice  makes 
perfect"  only  when  the  practice  is  carefully  planned  and  supervised  to  yield 
a  specific  result.  More  training  is  not  necessarily  the  answer,  better 
training  is. 


THE  HARDWARE  SUBSYSTEM 

Problems  with  equipment  are  evident  in  many  of  the  TEA.  These  problems 
include  equipment  malfunctions  and  nonavailability.  Reference  to  Table  L 
gives  an  overall  view  of  the  types  of  problems  found  and  how  these  problems 
impact  on  training.  Maintenance  problems  occur  most  often  with  new  equipment 
because  all  of  the  "bugs"  have  not  been  worked  out  ?nd  maintenance  personnel 
have  not  been  fully  trained.  For  fielded  systems,  the  problems  usually 
result  from  a  lack  of  repair  parts,  maintenance  equipment,  and/or  maintenance 
personnel . 


TABLE  L 

STUDIES  IN  WHICH  HARDWARE  PROBLEMS 
OCCURRED  WITH  MAJOR  IMPACT  ON  TRAINING 


STUDY 

PROBLEM 

RE  ARTS 

Lack  of  missiles  for  live  firing;  insufficient 
number  of  facilities  and  devices  to  support 
necessary  training 

PATRIOT 

Major  maintenance  problems;  lack  of  software 

CFV 

Target  devices  malfunctioned 

VULCAN 

Lack  of  maintenance  equipment;  equipment  failures; 
no  mounting  brackets  for  training  device 

CHAP/RE 

Lack  of  maintenance  personnel  and  repair  parts; 
insufficient  number  of  training  missiles 

MILES  AGES/AD 

MILES  equipment  malfunctions  caused  loss  of 
training  time  and  confidence  in  system 

MAGLAD/IRETS 

Range  mal functions 

IFV 

Nightsight  malfunction 

PA:  5.56mm 

Weapon  malfunctions  degraded  training 

Ml 

Equipment  inoperable 

Hardware  problems  also  are  related  to  design  deficiencies.  In  the 
VULCAN  study,  the  equipment  design  was  found  to  oppose  the  soldier's  natural 
tendency  to  correct  for  lag  in  tracking  a  target.  In  the  Ml  DVR  TNR  study, 
the  steering  indicator  was  ineffective  in  providing  proper  feedback  to  the 
trainee.  In  the  MLRS  report,  the  requirement  for  the  trainee  to  translate 
alphanumeric  codes  significantly  increased  mission  time.  These  are  problems 
in  the  area  of  human  factors  engineering  and  suggest  greater  attention  should 
be  given  to  the  soldier-hardware  interface. 
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THE  TRAINING  ENVIRONMENT 


Problems  associated  with  the  training  environment  are  numerous  and 
varied.  Too  often,  training  is  disrupted  by  work  details  or  putting  the 
soldier  in  a  support  role,  putting  on  VIP  shows,  or  similar  training 
distractors.  Training  facilities  often  are  inadequate.  This  includes 
crowded  classrooms  and  poor  environmental  conditions  (lack  of  air 
conditioning  as  noted  in  the  CHAP/RE  study  and  using  the  rear  tank  deck  for 
training  as  described  in  the  Ml  study).  Particularly  in  Europe,  adequate 
training  areas  are  limited  (RE  ARTS,  CHAP/RE,  PA:  MOUT,  PA:  5.55mm)  and  areas 
for  live  fire  training  are  restricted.  A  different  type  of  problem  was  noted 
in  the  NTSH  report.  Pilots  knew  the  military  reservation  over  which  they 
trained  so  well  that  the  training  was  "artificial"  and  contributed  little  to 
increasing  their  proficiency. 

To  summarize,  problem  areas  are  not  consistent  across  all  studies,  but 
the  problems  are  numerous.  Certain  problems  must  be  tolerated  to  some  extent 
(e.g.,  resource  and  cost  constraints).  Other  problems,  especially  the  low 
quality  of  training,  could  be  prevented  or  overcome  by  better  planning  from 
lessons  learned  and  a  greater  appreciation  of  the  importance  of  a  well- 
trained  and  highly  proficient  Army. 

WHAT  HAVE  WE  LEARNED  ABOUT  CONDUCTING  EVALUATIONS? 

The  TEA  Division  has  learned  many  lessons  as  a  result  of  the  development 
of  TEA  methods  and  additional  expertise  and  manpower.  In  addition,  the  TEA 
analysts  have  learned  that  certain  events  may  occur  that  cannot  be  prevented. 
Both  of  these  areas  are  presented  here  because  of  their  impact  on  conducting 
evaluations. 

LESSONS  LEARNED 

The  major  lessons  learned  by  the  Division  can  be  categorized  in  the 
following  areas: 

o  Coordinating  a  project 

o  Obtaining  data 

o  Data  analysis 

o  Feedback  to  the  Army 

Coordinating  a  Project 

The  experience  the  TEA  analysts  have  gained  over  the  past  four  years  is 
evident  in  the  initial  stages  of  a  TEA.  TEA  analysts  now  realize  what  can  or 
cannot  be  done  when  planning  a  project.  Sometimes  a  project  may  not  be 
undertaken  by  the  Division  because  what  is  proposed  by  a  proponent  is  not 
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feasible.  TEA  analysts  relying  on  accumulated  experience  have  thereby 
avoided  committing  valuable  resources  to  TEA  efforts  destined  to  be  plagued 
with  problems  and  unlikely  to  yield  useful  information. 

The  experience  and  expertise  of  the  TEA  personnel  also  have  resulted  in 
more  realistic  resource  estimates  and  answerable  objectives  for  studies  which 
are  undertaken.  The  documents  containing  these  estimates  and  objectives 
(Project  Coordination  Sheet,  Project  Plan  and  Study  Plan)  are  more  detailed, 
accurate,  and  relevant. 

Additionally,  the  TEA  analysts  have  become  more  sensitive  to  the  depth 
of  the  study  requirement  made  by  the  proponent.  The  more  intensive  the  TEA, 
the  more  manpower  needed  to  accomplish  the  objectives.  The  Division  expends 
3-5  man  years  to  complete  most  major  TEA. 

Obtaining  Data 

In  many  TEA,  it  is  necessary  to  obtain  background  data,  course  records, 
and  standardized  test  scores  from  other  Army  agencies.  The  personal  contacts 
made  with  the  proponents,  MILPERCEN,  and  DMDC  have  improved  the  acquisition 
of  these  data.  Personnel  of  the  TEA  Division  realize  what  data  are  needed, 
where  they  can  be  obtained,  and  the  approximate  length  of  time  required  to 
receive  the  data.  These  lessons  have  proven  very  beneficial  in  estimating 
cost  and  manpower  for  a  project  and  in  the  execution  of  the  TEA  by  allowing 
the  obtained  data  to  be  related  to  the  results  of  tests  and  surveys 
administered  by  the  Division.  TEA  projects  now  bring  together  a  wealth  of 
information  from  various  sources.  The  result  is  a  more  thorough  and  accurate 
picture  of  Army  training. 

Data  Analysis 

With  the  added  expertise  of  mathematicians,  statisticians,  and  computer 
scientists,  data  analysis  plans  now  are  more  detailed,  and  full  advantage  is 
made  of  the  UNIVAC  1100/82  computer.  Analysts  frequently  use  the  package 
programs  available  on  the  computer,  particularly  those  of  the  Statistical 
Package  for  the  Social  Sciences  (SPSS)  and  the  Biomedical  Computer  Programs 
(BMDP).  Large  amounts  of  data  are  now  analyzed  in  a  shorter  period  of  time 
with  a  great  savings  in  manpower.  The  SPSS  and  BMDP  packages  not  only 
include  the  more  common  parametric  statistical  tests  but  also  most  of  the 
nonparametric  procedures  for  analyzing  data  from  nominal  and  ordinal 
measurement  and/or  small  samples. 

Feedback  to  the  Army 

It  is  TEA  Division  policy  to  brief  the  results  of  studies  to  the 
proponents  and  to  the  tactical  field  units  where  the  data  were  collected. 

The  briefing  to  the  proponent  is  the  first  presentation  of  the  results  of  the 
TEA.  For  the  field  units,  these  briefings  provide  feedback  on  training 
deficiencies  and  other  findings  of  the  study.  From  this  information,  the 
unit  commander  can  identify  strengths  and  weaknesses  of  the  unit  training 
programs.  For  the  Division,  the  result  of  these  briefings  is  to  increase  the 
visibility  of  the  TEA  process. 
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TRADOC  schools  and  centers  which  have  not  been  briefed  on  the  TEA 
process  or  a  TEA  study  are  encouraged  to  request  a  formal  presentation. 

These  briefings  are  an  integral  part  of  the  TEA  process  and  provide  a  means 
of  presenting  TEA  results  in  a  clear,  concise,  and  interesting  manner. 

EVENTS/ PROBLEMS  ENCOUNTERED  IN  CONDUCTING  EVALUATIONS 

In  the  conduct  of  TEA  certain  problems  have  been  encountered  which  are 
beyond  the  control  of  the  analysts.  These  problems  are  documented  here  to 
provide  the  newcomer  to  the  TEA  process  an  overview  of  possible  pitfalls  In 
accomplishing  a  TEA.  These  problems  are  specifically  related  to: 

o  Samples 

o  Scheduling 

o  Collecting  data  on  a  non-interfering  basis 

o  Incompleteness  of  data 

o  Resource  constraints 

Ideally,  the  hope  of  any  analyst  is  to  obtain  a  representative  sample  of 
adequate  size.  Sample  size  depends  on  the  developmental  phase  of  the  system 
under  study,  availability  of  the  appropriate  soldiers,  and  cost  of  collecting 
data  (e.g.,  travel  costs).  The  impact  of  a  study  is  often  dependent  on  which 
and  how  many  soldiers  were  participants. 

It  is  often  difficult  to  adhere  to  data  collection  schedules  and 
conditions.  Soldiers  often  will  not  be  available  at  the  requested  time. 

When  scheduling  data  collection  activities,  especially  hands-on  tests,  more 
than  one  day  should  be  set  aside  for  the  personnel  and  equipment  to  be 
available  for  testing.  In  reality,  this  is  rarely  possible.  The  one  day 
scheduled  for  the  test  will  probably  be  the  only  day  the  test  soldiers  will 
be  available.  If  something  happens  on  that  day  to  degrade  the  results  (see 
the  MAGLAD/IRETS  and  Ml  UCOFT  reports),  the  choice  is  to  "live  with  it"  and 
interpret  the  results  accordingly  or  not  complete  the  project. 

In  several  of  the  TEA  studies  (IFV,  PATRIOT,  PERSHING,  MLRS) ,  the 
Division  analysts  agreed  to  collect  data  on  a  non-interfering  basis,  i.e., 
without  interrupting  the  ongoing  activities  of  a  unit  or  training  program. 
This  sometimes  results  in  performing  subjective  evaluations,  or  "educated 
guesses",  about  performance.  In  the  NTSH  project,  timely  and  adequate  access 
to  base  case  hardware  and  alternatives  was  not  given. 

The  first  source  of  incomplete  Information  is  associated  with  data 
collected  from  other  sources.  Course  records  often  do  not  Include  specific 
reasons  for  a  soldier  failing  to  graduate  from  a  class.  Standardized  scores 
and  demographic  data  obtained  from  the  school  and  other  sources  often  are 
incomplete  or  only  In  summary  form  which  limits  usefulness.  In  some 
Instances,  no  records  are  available  for  certain  individuals. 
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The  second  source  of  incomplete  information  is  related  to  equipment 
malfunctions  or  design  problems.  Data  collection  efforts  may  be  delayed 
because  of  maintenance  difficulties.  During  the  CFV  study,  one  target  device 
malfunctioned  and  another  had  to  be  substituted.  This  type  of  problem  causes 
the  test  plan  to  be  altered  and  data  collection  to  be  delayed.  The  UH60FS 
TDS  was  delayed  for  over  six  months  due  to  temperature  and  humidity  effects 
on  the  computers.  In  the  PATRIOT  study,  the  print-out  score  from  the  Troop 
Proficiency  Trainer  (TPT)  was  found  to  be  an  inaccurate  reflection  of  soldier 
performance.  In  the  ELSAP  project,  performance  proficiency  tests  were  nof 
possible  because  the  Swiss  Army  could  not  provide  a  tank  as  planned.  These 
types  of  problems  often  will  be  encountered  when  relying  on  equipment  to 
provide  assessment  data.  Similar  kinds  of  problems  develop  when  analysts 
must  use  unapproved  task  lists  as  in  new  equipment  training  evaluations.  The 
results  of  the  TEA  may  be  limited  when  the  approved  task  list  is  published. 
Often,  the  Division  must  compare  a  system  to  conceptual  alternatives  that  do 
not  exist  in  the  real  world. 

There  are  certain  instances  in  which  the  Division  requests  sped  r;r 
samples,  date,  for  collecting  data,  cost  information,  or  equipment,  and  t^  v 
requests  are  not  met.  The  reasons  for  not  meeting  these  requests  often  are 
associated  with  the  proponent's  mission  requirements,  inability  to  comply 
because  of  deadlines  to  be  met,  or  the  cost  of  fulfilling  a  request. 

Since  certain  objectives  of  the  TEA  are  dependent  on  these  requests  being 
met,  it  may  be  necessary  to  renegotiate  a  Study  Plan,  Project  Plan,  or 
Project  Coordination  Sheet.  It  may  happen  that  these  problems  are  not  known 
until  the  study  is  nearly  completed.  Because  of  resource  and  schedule 
limitations  external  to  the  Division  and  commitments  to  other  proponents,  the 
study  will  be  completed,  and  the  reasons  for  not  meeting  certain  objectives 
will  be  documented  in  the  report. 

The  lessons  learned  by  the  Division  provide  a  firm  foundation  for  future 
development  of  the  TEA  process.  As  a  result  of  past  efforts.  Division 
analysts  are  better  able  to  recognize  what  information  is  required  for  a 
particular  study  and  anticipate  the  nature  and  impact  of  problems  likely  to 
be  encountered  In  collecting  that  information.  Division  analysts  have  become 
more  highly  skilled  in  obtaining  and  analyzing  data.  As  the  TEA  process 
grows  and  develops,  it  will  be  necessary  to  build  on  what  has  been  learned. 
The  continued  evolution  of  TEA  methodology  requires  further  research, 
refinement  of  test  development  procedures,  and  adoption  of  more  sophisticated 
analytical  techniques.  The  methods  and  procedures  which  have  served  well  in 
the  past  may  need  to  be  improved  in  the  future. 

WHERE  ARE  WE  HEADED? 

When  established  in  1978,  the  goals  of  the  Division  were  to: 

o  Contribute  to  the  fielding  of  optimum  training  packages  with  the 
hardware 

o  Assist  in  the  improvement  of  existing  training  packages 


o  Build  a  TEA  data  base 


These  remain  the  primary  goals  of  the  Division  today.  However,  special 
emphases  within  these  goals  have  developed  as  a  result  of  the  added 
experience  of  the  Division  and  changes  in  Army  structure  and  organization. 
These  emphases  include: 

o  Army  Force  Modernization 

oo  Conducting  more  TEA  on  post-fielded  systems 
oo  Conducting  TEA  early  in  the  developmental  phase 

o  Permeation  of  the  TEA  process  throughout  the  Army 

The  future  directions  of  the  TEA  process  will  be  strongly  influenced  by  the 
Division's  attempt  to  meet  these  objectives. 

ARMY  FORCE  MODERNIZATION 

Development  of  TEA  Procedures  for  Post-Fielded  Systems 

Currently,  the  Division  is  fine-tuning  TEA  procedures  for  post-fielded 
systems  which  will  allow  more  immediate  feedback  to  unit  commanders.  The 
focus  of  these  TEA  will  be  on  soldier  proficiency  in  those  tasks  and 
functions  essential  to  success  in  combat.  To  accelerate  the  TEA  process,  the 
analytical  procedures  will  be  less  sophisticated  statistically,  more 
subjective  in  nature,  and  rely  more  on  existing  test  instruments  than  full- 
scale  TEA.  Although  more  narrow  in  scope  than  major  long-term  studies,  these 
TEA  will  include  hands-on  tests,  skills  and  knowledge  tests,  assessment  of 
soldier's  attitudes  toward  training,  and  examination  of  SQT  results  and  unit 
evaluation  exercises.  The  result  of  such  limited-scope  TEAs  will  be  more 
general  in  nature  than  detailed  studies  and  will  focus  on  broad  issues  and 
yield  trends  and  hypotheses  that  will  help  identify  systems  requiring  more 
in-depth  analysis. 

To  support  an  increase  in  post-fielded  system  analyses,  the  Division  has 
proposed  that  a  TEA  cell  be  established  in  Europe.  The  cell  would  allow  the 
TEA  analysts  to  be  "on-site"  with  units  stationed  in  Europe.  The  result 
would  be  more  timely  analyses  tailored  to  the  specific  training  needs  of 
individual  units  in  their  operational  environment. 

Completing  TEAs  Early  in  the  Developmental  Phase 

The  impact  of  force  modernization  will  be  widespread  and  include  the  TEA 
Division.  TEA  will  have  to  be  conducted  very  early  in  the  developmental 
phases  of  new  equipment,  perhaps  even  during  the  conceptual  phase,  to  insure 
the  most  efficient  systems  are  fielded. 

The  Division  has  already  become  involved  in  force  modernization  problems 
with  the  "Corps  Support  Weapon  System  (CSWS)  Preliminary  TEA."  The  TEA 
compared  five  alternatives  which  have  been  proposed  to  serve  as  the  corps 
support  weapon  system.  The  comparison  included  personnel  resource 
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requirements,  training  support  requi rements ,  behavioral  requirements  of 
equipment-related  tasks,  and  soldier  perceptions  of  training  adequacy.  Since 
only  two  of  the  alternatives  have  been  fielded,  certain  aspects  of  the 
comparison  were  conceptual. 

The  CSWS  study  is  important  because  it  is  the  first  TEA  conducted  by  the 
Division  on  conceptual  systems.  Another  evaluation  of  a  conceptual  system, 
the  Direct  Support  Automatic  Test  Support  System,  is  already  underway. 
Evaluations  of  conceptual  systems  will  provide  valuable  information  tc  ':ys*<v 
contractors,  combat  force  developers  and  training  developers  about  systen 
requirements  in  the  areas  of  personnel,  training,  and  equipment  design. 

PERMEATION  OF  THE  TEA  PROCESS 

The  TEA  Management  Agent  is  charged  with  promoting  the  TEA  process 
throughout  TRADOC  and  the  Army.  To  date,  studies  have  been  completed  for 
eleven  different  TRADOC  schools  and  centers.  Additional  studies  ire 
presently  underway,  including  some  for  proponents  which  have  not  previous':; 
requested  a  TEA.  Any  TRADOC  school  or  center  which  has  not  requested  the 
Division's  assistance  is  encouraged  to  do  so.  The  Division  hopes  to  make  i.f 
TEA  process  an  integral  part  of  all  Army  training. 

In  an  effort  to  promote  the  TEA  process,  the  TEA  methodology  will  lit 
updated.  Specific  ideas  about  the  conduct  of  TEAs  have  been  proposed  since 
the  draft  TEA  Handbook  was  published  in  1979.  This  handbook  gave  only 
general  guidelines  to  accomplish  a  CTEA.  A  pamphlet  will  be  produced  in  the 
future  updating  this  CTEA  methodology.  In  addition,  the  handbook  will  also 
be  revised.  Both  of  these  documents  should  prove  beneficial  to  TEA  analysts 
and  to  proponents. 

The  TEA  Division  will  assist  the  TEA  Management  Agent  in  conducting 
seminars  in  TEA  procedures  as  yet  another  means  of  promoting  the  TEA  process. 
The  first  seminar,  "Introduction  to  Training  Effectiveness  Analysis  (TEA)," 
was  held  20  September  through  1  October  1982,  at  the  US  Army  Air  Defense 
Center,  Fort  Bliss,  Texas.  The  course  was  directed  toward  military  and 
civilian  action  officers  whose  duties  require  them  to  supervise,  plan, 
conduct,  monitor,  and/or  review  TEA  studies.  The  seminar  emphasized  TEA 
policies,  philosophy,  principles,  processes,  and  methods.  The  training 
focused  on  the  "how  to"  aspects  of  the  TEA  process  and  the  rationale 
underlying  TEA  methods.  The  course  provided  the  action  officer  an 
opportunity  to  expand  his  or  her  abilities  with  subsequent  on-the-job 
application  of  the  TEA  process,  TEA  study  experience,  and  additional 
professional  training  to  become  a  fully  qualified  TEA  action  officer. 
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SUMMARY 


In  reviewing  the  activities  of  the  TEA  Management  Agent  and  the  TEA 
Division  since  its  inception,  it  is  obvious  that  much  has  been  done  and  much 
has  been  learned.  Over  thirty  major  studies  have  been  accomplished  in 
support  of  eleven  different  proponents.  These  studies  have  covered  a  wide 
array  of  topics,  from  the  evaluation  of  tanks  to  flight  simulators,  from 
rifles  to  missiles.  The  results  of  these  studies  have  contributed  to 
decisions  on  whether  a  system  should  be  fielded,  how  training  programs  should 
be  conducted,  how  trainees  should  be  selected,  and  how  equipment  should  be 
designed  to  maximize  soldier  proficiency.  Deficiencies  in  Army  training  have 
been  identified  and  solutions  recommended. 

In  the  process  of  evaluating  training  programs,  the  methods  and 
techniques  of  the  analysis  process  itself  have  been  evaluated.  As  a  result, 
training  effectiveness  analytical  procedures  are  being  modified  to  insure  the 
data  collected  are  reliable,  valid,  and  properly  analyzed  and  interpreted. 

In  the  next  few  years,  new  demands  will  be  placed  on  the  TEA  process  as 
the  Army  modernizes  its  forces  and  as  more  proponents  request  assistance  from 
the  Division.  To  meet  these  demands,  the  TEA  process  must  be  continuously 
reviewed  and  modified  as  required.  It  must  remain  a  process  in  evolution. 
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LISTING  2 

STUDIES  INCLUDING  A 
S/K  TEST  DEVELOPED  BY  TEA  DIVISION 
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Listing  2  (Continued) 


Study  Soldiers  Numbers  of  Items 


Ml 

19  K/L 

40 

45E 

40 

63E 

40 

Ml  UCOFT 

UCOFT 

40 

90 


Ml  DVR  TRN  CREW  40 

PERSHING  15E  150 

21G  63 

214E0  69 

FV  UCOFT  20 


Percent  of 

Items  Answered  Correctly 


19  K/L  -  66% 
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LISTING  3 

STUDIES  INCLUDING  GENERAL  ATTITUDE  SURVEYS 
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LISTING  4 

STUDIES  INCLUDING  TASK  SURVEYS 
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LISTING  6 

STATISTICS  USED  IN  STUDIES  TO 
DETERMINE  EXTENT  TWO  OR  MORE  VARIABLES  ARE  RELATED 
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